MULTI-SENSOR INFORMATION FUSION TECHNIQUE 

Field of the Invention 

The present invention is related generally to a method and system for locating 
the position of, and tracking, objects. More specifically, the invention is directed 
towards a method and system for locating and tracking airborne aircraft. 

Background of the Invention 

In recent years, there has been a steady increase in the number of airplanes in 
service at any one time. With this rise in the number of aircraft, there has been an 
increased interest in minimizing the separation distance between airplanes without 
compromising safety. Additionally, there is a trend towards autonomous or free-flight 
planning wherein the pilot, while en route, will be permitted to change the aircraft's 
flight plan. In order to accomplish these and other objectives, it is desirable for air 
traffic controllers and pilots to know, as precisely as possible, the location of all 
airplanes relative to one another. 

Typically, several sensor systems including radar systems, beacon systems, 
etc., are employed for surveillance and tracking, along with current position reports 
from pilots, estimations based on flight plans and vehicle speed, etc. These methods 
are imprecise and limited in that they individually report different approximate 
positions for the same object such as an airborne aircraft, and a margin of error must 
be applied to each reported location. These differences in the reported locations and 
errors arise because of the performance and operational characteristics of the sensors, 
and the prevailing environmental conditions. For example, the reliability of 
information from a radar system, no matter how sophisticated, might vary greatly with 



atmospheric conditions, most likely being more precise under clear and calm 
conditions than under stormy conditions (rain, snow, hail, etc.). The margin of error, 
when applied to each quantitative and qualitative observation, results in the tracking 
system applying a larger than desirable separation zone around the best estimated 
aircraft location. This separation zone, or uncertainty zone, is typically increased 
during foul weather in order to compensate for the performance characteristics of the 
sensors and the less than optimal reliability and accuracy of the data. 

Such existing methods and systems for locating an object are generally 
directed towards guiding an aircraft onto the runway when landing. Other systems, 
such as the Traffic Alert and Collision Avoidance System (TCAS) issue alerts and 
evasive commands only after the safety zone has been breached. The TCAS relies on 
data transmitted by a transponder located on an aircraft. Since all aircraft are not 
mandated to have a transponder, the system can not be relied upon for maintaining 
minimum and safe separation zones. Other current methods collect data from several 
sources, and then report the location information either by applying a weighted 
average to the data, or relaying the data from a single sensor, or transmitting all 
information as separate attributes (e.g., speed, direction, altitude, etc.). 

Therefore, there exists a need for a method and apparatus to collect ■all- 
available quantitative and qualitative data from the numerous sources and aggregate 
these with prior knowledge of their performance characteristics to more accurately 
locate and track an object such as an airborne aircraft. 

Summary of the Invention 

The disclosed invention is a method and apparatus for aggregating, or fusing, 

information from several sources into one or more reports which more accurately 
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locates the position of an object such as an airborne aircraft and tracks its path. In the 
preferred embodiment of the present invention, statistical analysis and probability 
theory are preferably used to combine quantitative data (i.e., numeric data from 
sensors) and qualitative information, e.g., symbolic, weather conditions, reliability of 
5 source, assumptions, etc., to derive the object's position with a higher level of 
confidence than can be attained using one or more sensors either separately or with 
the combination methods and systems of the prior art. This accuracy, achievable with 
the present invention, is preferably attained by segmenting all possible attributes 
reported by the different sensors and applying one or more statistical operators to 
10 derive the intersection of all reports to generate a value indicative of the most likely 
location of the object. 

Brief Description of the Drawings 
Figure 1 is a conceptual drawing illustrating an object locator in accordance 
with the present invention; 
15 Figure 2 shows the several conflicting position reports provided by various 

sensors; 

Figure 3 is a schematic diagram showing an aggregated position report 
generated by an illustrative embodiment of the present invention; 

Figure 4 is a chart illustrating the probability distributions of an object's 
20 location segmented into sub-ranges or classes; and 

Figure 5 is a schematic diagram showing a fuzzy logic neural network that 
may be used for analyzing qualitative data in accordance with the present invention. 

Detailed Description of the Preferred Embodiments 
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Figure 1 is a conceptual drawing illustrating an object locator 10 in accordance 
with the present invention. Object locator 10 preferably fuses (or aggregates) inputs 
from a plurality of sensors and/or sources 12, 16, 20, and 24 for locating and tracking 
the position of an object such as an airborne aircraft. In the illustrative embodiment, 
sensors 12 and 20 may provide quantitative data and sources 16 and 24 may provide 
qualitative information. Hereinafter, the term sensor is used to collectively represent 
sensors and/or sources. The four sensors 12, 16, 20, and 24 shown in Figure 1 are for 
illustration purposes only, and any number of sensors can be accommodated by the 
present invention. Sensors 12, 16, 20, and 24 scan for objects 1, 2, 3, 4, and 5, and 
provide information regarding the estimated positions of these objects to the object 
locator 10 along paths 14, 18, 22, and 26, respectively. The five objects 1, 2, 3, 4, and 
5 shown in Figure 1 are for illustration purposes only, and any number of objects can 
be accommodated by the present invention. 

Information regarding the qualitative and quantitative performance 
characteristics of each sensor (and/or source) is also available to object locator 10. 
Typical sensor performance characteristics include information on the sensor's 
accuracy, reliability, effect of environmental conditions (e.g., temperature, pressure, 
precipitation, etc.), degradation with age, etc. This information may be stored in a 
database 28 accessible by the object locator 10 along path 29. Alternately, each 
sensor 12, 16, 20, and 24 may transmit its performance characteristic along with the 
estimated positions of the objects on its respective path 14, 18, 22, and 26 to object 
locator 10. Object locator 10 preferably uses statistical methods to conduct a 
probability analysis on the data to generate a value indicative of the most likely 
location of the object. 
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Since each sensor may have its own unique performance characteristic and in 
many cases may be located distant from other sensors, object locator 10 may receive 
conflicting reports on the estimated position of the same identified object. 
Additionally, each reported position will likely have a level of uncertainty associated 
with it reflective of the performance characteristics of the corresponding sensor or 
sensors. This is illustrated in Figure 2 wherein the four sensors of Figure 1 are shown 
reporting four distinctly different and conflicting positions and uncertainty levels for 
the same object 30. For instance, one of the four sensors is shown as reporting object 
30 traversing along path 38 and located at the center of the sensor's uncertainty 
boundary 32. Concurrently, a second sensor is shown as reporting object 30 
traversing along path 48 and positioned at the center of the sensor's uncertainty 
boundary 42. A third sensor is shown as reporting object 30 traversing along path 58 
and located at the center of the sensor's uncertainty boundary 52. Finally, a fourth 
sensor is shown as reporting object 30 traversing along path 68 and positioned at the 
center of the sensor's uncertainty boundary 62. 

Under the prior art method, the reported locations and uncertainty boundaries 

may be aggregated using weighted averaging technique and object 30 may be reported 

as being located at the center of uncertainty boundary 72. Subsequently, a safety zone 

equivalent in size to uncertainty boundary 72 may be designated around object 30, 

and no other object is permitted to encroach into this region. Uncertainty boundary 

72 encompasses, and hence may be relatively larger than, the individual sensor 

uncertainty boundaries 32, 42, 52, and 62. Additionally, data from only some of the 

available sensors may be taken into consideration. Furthermore, only a small subset 

of all the data available from the sensors under consideration may be analyzed. As 
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discussed hereinafter, these shortcomings of the prior art may be reduced or 
eliminated in the preferred embodiment of the present invention. 

Figure 3 is a schematic representation of a most likely position of object 30 as 
derived by an illustrative embodiment of the present invention. Details for locating 
and tracking the object are presented later in discussions related to Figures 4 and 5. In 
general, the preferred embodiment divides the uncertainty boundary of each sensor 
into a plurality of segments such as 33, 34, and 35 associated with a first sensor; 43, 
44, and 45 associated with a second sensor; 53, 54, and 55 associated with a third 
sensor; 63. 64, and 65 associated with a fourth sensor; and so on. These uncertainty 
boundaries for the object are propagated and decimated into a plurality of finer 
overlapping segments. As discussed below, each segment 33, 54, 65, etc., may be 
weighted with a probability measure, and a statistical combinatory approach is then 
preferably applied to more accurately identify the most likely position with a smaller 
uncertainty region 82 for object 30. This is one advantage of the preferred 
embodiment of the present invention. Another advantage is that an estimate of the 
path 88 being traversed by object 30, and the associated path uncertainty 89, may be 
more accurately determined. 

For each sensor available for locating and tracking an object, a probability 

distribution may be determined based on the reported estimated position of the object 

and the performance characteristics of the sensor. As shown in Figure 4, example 

curve 412 may represent the probability distribution of the estimated position of the 

object as determined by sensor 12. Likewise, example curve 416 may be the 

probability distribution of the estimated position of the object as determined by sensor 

16. Example curve 420 may indicate the probability distribution of the estimated 
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position of the object as determined by sensor 20, and example curve 424 may 
represent the probability distribution of the estimated position of the object as 
determined by sensor 24. As before, the four probability distributions corresponding 
to the four sensors are merely illustrative in nature. 

The probability distributions are preferably segmented into sub-ranges or 
classes cj 401, C2 402, cj 403, etc. This segmentation results in a plurality of 
uncertainty boundaries for the estimated position of the object, and a probability value 
is preferably associated with each sub-range. It should be noted that for each sensor, 
every sub-range such as those illustrated in Figure 4 may be represented by an 
uncertainty boundary in Figure 3. Again, for illustration purposes, only three of the 
six sub-ranges from Figure 4 are shown as uncertainty boundaries in Figure 3 for each 
of the four illustrated sensors. It is desirable that the intersections of the plurality of 
segments result in new boundaries that indicate the probable area wherein the object 
may be located. These new boundaries may then be statistically evaluated to identify 
the class with the greatest likelihood of containing the object. 

The probability analysis may be further extended to combine the quantitative 

data and qualitative information (e.g., symbolic, weather conditions, reliability of 

source, assumptions, performance history, etc.) to provide more accuracy in locating 

and tracking the object. For each identified sub-range of Figure 4, all imposed 

assumptions and quantitative and qualitative indicators may be taken into 

consideration, and a probability measure reflective of the reliability of the sensor to 

sense the information is preferably assigned. In the prior art, a sensor's overall 

historic performance data may be extrapolated to obtain a trending indication of the 

sensor's reliability. However, in the preferred embodiment of the present invention, 
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the extrapolation is preferably applied to each of the parameters (e.g., temperature, 
pressure, moisture level, visibility, etc.) that impact a sensor's reliability. By 
individually extrapolating on each parameter affecting a sensor's reliability, the 
overall estimation error may be reduced. An advantage of this approach is that it 
permits the combining of any and all types and forms of variables and parameters, 
collectively referred to as drivers, that can impact a sensor's reliability. 

As shown in Figure 5, representative examples of drivers includes fixed 
known values, probable ranges 504 and 510, fuzzy logic based rules on qualitative 
environmental conditions 508, fuzzy logic based rules on assumptions effecting 
sensor's operational performance 502, functional expressions 506, etc. Known 
quantities may be specified as fixed values and uncertain parameters are preferably 
represented by probability distributions such as 504 and 510. The probability 
distribution 504, for example, may be representative of a best curve fit of historic data 
from the sensor, and the fuzzy rules on conditions 508, for example, may include 
qualitative information regarding the effect of environmental conditions (e.g., rain) on 
the accuracy of a reading from a sensor. 

In one illustrative embodiment of the present invention shown in Figure 5, all 

the defined, drivers are preferably input to a neural network with multi-level 

perceptron 520 for estimating the nonlinear relationship between the driver's values 

and the sensor's reliability. Neural network 520 may be trained to determine the 

reliability measure, P[s], and the realization measure P[siJ 530, preferably indicative 

of the mean of the sensor reliability measure for a given class. Next, a simulation 

such as a Monte-Carlo approach for example, may be used to determine the statistical 

distribution for all possible ranges with the likelihood outcome for the senor 
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reliability, and the conditional probability rule is preferably applied to compute the 
posteriori conditional probability distribution for each class c z 

p\ c j Sk l.ffcufj Equation 1 

l ' kl P[s k ] 

where, 

5 P[c f /s k ] is the conditional probability of class c, given that the measurement 

is from sensor k 

P[c n s k ] is the intersection probability of sensor reading s k residing in class c. 

Lastly, a conjunctive fusion method, which provides an estimation of the 
intersections of the probability measures, is preferably applied to all parameters 
10 affecting sensor reliability to identify the class with the most likely probability of 
defining an object's position. For instance, an information class, c io , may be depicted 
such that 



c io = arg max 



f(P[c n s { ],..., P[c n s N ]) Equation 2 

max [f(P[Cj, Sl ],..., P[c ; ,^])]| 



where, in the ratio on the right hand side, the function in the numerator represents an 
15 intersection operator which takes into account the probability of all sensor readings, sj 
through sn, located in class c„ and the denominator is the maximum value of the 
intersection operator from all the classes. 

The ratio on the right hand side of Equation 2 is preferably evaluated for each 
class, and the final attribute comes from the depicted class that maximizes the 
20 argument. All the qualitative information and assumptions are preferably included as 
part of this analysis, which strengthens its capabilities in possibly providing a better 

-9- 



estimate of the object's position. In the event that sensor reliability may not be 
included (i.e., qualitative information is not fused), or may be assumed to be the same 
for all sensors, the conjunctive fusion method simplifies to: 



c io = argmax 



/(P[C | /5 1 ] V .. > P[C,/^]) 

m^f^PiCj/s^PiCj/s^ 



Equation 3 



The foregoing discussion focused on a method and system of the preferred 
embodiment of the present invention for locating and tracking an object, such as an 
airborne aircraft, preferably using statistical analysis and probability theory for fusing 
(or aggregating) quantitative data and qualitative information from a plurality of 
sensors and sources. In considering alternate embodiments of the present invention, it 
may be convenient to refer to the above described preferred embodiment as a local 
fusion center or alternately as a local object locator. In another embodiment, a 
plurality of local fusion centers may be considered as a plurality of sensors reporting 
information on the position and probability distribution for a plurality of objects. In 
such an embodiment, the plurality of local object locators may transmit the plurality 
of object locations and probability distributions to a central (or global) fusion center 
for further locating and tracking of objects on a global basis. 

In one illustrative embodiment, a site such as an airport, can be considered as 
having one or more local fusion centers for locating and tracking not only airborne 
aircraft, but also other objects such as service vehicles and aircraft on the tarmac. The 
plurality of local object locators at a site may be considered as providing location and 
probability distribution information to a site fusion center for that airport, which in 
turn may transmit relevant data to yet another fusion center possibly receiving data 
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from other site fusion centers. Preferably, the lowest-most local fusion center may 
either transmit the fused data including object positions and probability distributions, 
and/or all the quantitative data and qualitative information received from its 
associated sensors. 

5 Numerous advantages of the invention covered by this document have been 

set forth in the foregoing description. It will be understood, however, that this 
disclosure is, in many respects, only illustrative. Changes may be made in details, 
particularly in matters of shape, size, and arrangement of parts without exceeding the 
scope of the invention. The invention's scope is, of course, defined in the language in 

10 which the appended claims are expressed. 



-11- 



